Automation of testing holds great promise for testing products in an efficient and reliable manner. Ideally, test routines would automatically execute in an arbitrary sequence without the need for human supervision. However, the reality of test automation has fallen far short of its promise.
Automated testing may seek to test a variety of testing protocols, such as protocols randomly selected or a variety of protocols. However, automated testing is typically limited in the testing protocols selected. For example, the selection of protocols for testing may be limited to those protocols which have a common starting point. Otherwise, if protocols are selected without a common starting point, manual intervention and/or rebooting of the system may be required to run the testing.
Moreover, automated testing searches for exceptional occurrences in the system, such as failures in the system. When previous automated testing schemes encountered an exceptional circumstance, they were unable to address it in an effective manner. Previous automated systems might simply halt, requiring manual intervention by an operator. The manual intervention may include a rebooting of the system. If rebooting is unsuccessful, the operator might then reinstall the software to fix the problem. This dependence on operator intervention is costly from both a time and a human perspective. The testing software may sit idly for a considerable amount of time before an operator may intervene to fix the problem. Further, the time required to reinstall the software may be considerable. Other automated systems have attempted to compensate for an exceptional occurrence by causing, in effect, a restart or a reboot of the testing system. All of the routines necessary to execute the testing system are reloaded, thereby reinitializing the system. The reloading of the system software may enable the test automation to recover after a fatal error. However, this solution is again inadequate since rebooting is a brute force method which affects all of the system, including those portions of the system which are operating correctly. The time involved in reloading the system software can be significant, especially with systems that have large files that must be reloaded into RAM. Further, rebooting the system may destroy valuable information for diagnosing the exceptional occurrence.
What is needed is a better method and system for automated testing.